{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "基于A2C算法实现,A2C就是去基线的AC算法.\n",
    "\n",
    "因为有通信,所以所有actor的视野是共享的,它们的state相同.\n",
    "\n",
    "因为是合作任务,reward也相同.\n",
    "\n",
    "因为有通信,所以只需要一个critic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAR8AAAEYCAYAAABlUvL1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAnN0lEQVR4nO3df1xUdb4/8Nc5MzCCMINgzMAKSqu7yvobDWe1b21yJWMrk2+txprXvHKX0FLL3eU+zFZvG67truW24t26K3ZvXrs8vg/NXH/EYqnpiIpRiESYdqFkoCRmQGWAOZ/vH15OTZo6AvNh8vV8PM7jIefznvl8zpHzehzOZ+YcRQghQEQUYKrsARDRzYnhQ0RSMHyISAqGDxFJwfAhIikYPkQkBcOHiKRg+BCRFAwfIpKC4UNEUkgLnz//+c8YMmQI+vXrh9TUVBw5ckTWUIhIAinh8/rrr2Pp0qV45plncPz4cYwZMwbp6elobGyUMRwikkCR8cXS1NRUTJw4ES+99BIAQNM0JCQkYNGiRfj1r38d6OEQkQTGQHfY3t6OsrIy5OXl6etUVUVaWhocDscVX+PxeODxePSfNU1DU1MTYmJioChKr4+ZiK6PEAItLS2Ij4+Hql79D6uAh88XX3wBr9cLq9Xqs95qteLDDz+84mvy8/OxcuXKQAyPiHpAXV0dBg0adNWagIfPjcjLy8PSpUv1n10uFxITE1FXVwez2SxxZETdJ4RAQ8PnePXVEhw8KKCqYT343hpiYpowf/44TJo0/ppnI93ldruRkJCAyMjIa9YGPHwGDhwIg8GAhoYGn/UNDQ2w2WxXfI3JZILJZLpsvdlsZvhQ0Lt4sQ1r1+5CTc0AhIZe/nveXS5XOP74xxN49tloTJw4JiCXKq6nj4DPdoWGhiIlJQUlJSX6Ok3TUFJSArvdHujhEEnl8bSjoGA7Tp0yw2Do+eABAEVRAVjx4ouHcPp0ba/0cSOkTLUvXboUL7/8MjZt2oSqqirk5OTg/PnzmDdvnozhEEkhhEBJiQN/+1sTVDW8V/tSFBWffx6NF1/8G86fv9CrfV0vKdd8fvazn+Hzzz/HihUr4HQ6MXbsWOzevfuyi9BE32VnzzrxyivHYTB8LyD9KYoBlZWheOONdzBr1t29fv3nmuMJxhvIu91uWCwWuFwuXvOhoKRpGlat+isOHgyHqoYErF8hBIzGOmzY8DMkJl59NupG+HNs8rtdRBLU1tbB4XAFNHiASxeC29tjsX37Acg+72D4EAWYEAKHDn2Azs5oKf2rqgmHD3/q88FdKeOQ2jvRTUjTNOzffwoGQ+9eZP42iqKgvj4ENTWnpfTfheFDFGCffvoZTp/2/u8UuByKEoP9+9+X1j/A8CEKuHPnmtHREdhrPd+kqqFwOl3QNE3eGKT1THSTKiv7EAaDnOs9X1FQWfml1Os+DB+iAPN6NQBy78agKAo0jbNdRHQTYvgQkRQMH6IAS0qKg6adlzoGITR873v9YTTKu6sOw4cowJKS4iGE7PARGDQoAiEh8mbdGD5EATZo0PdgtXZK/XpDZ+eXmDRpuLT+AYYPUcCFhYXhttvioGkdUvoXQiAy0o2xY38kpf8uDB+iAFMUBXfeOQaa9oWkEWgYMyYKFovcO0IwfIgkGDFiGG69VUCIwH/CWNO+QEbGROn382H4EEnQr18/PPLIZKhqU0D79XrbMWlSGFJSRl2z9uLFizhy5Aj279+P48ePo729vUfHEhRPryD6rlEUBZMnj8ddd32Et966CIOh555Y8W2EEIiK+gLZ2ZlXneXq6OjA22+/jcLCQlRVVcHr9cJoNGL8+PGYN28e7HZ7j5w18cyHSBKDwYB589IxaFALhPD2al9CCKhqM+bNG4uEhG+/bWtHRwdefvll5OXl4cMPP4SqqggJCYGiKDh+/DiWLFmCoqKiHvlCKsOHSKKBA2OwatUMhIfX9+rUu9fbgqyseEyffvtVH2tz8OBB/Pu//zuEEJfVKYqCzs5OrF27FidPnuz2mBg+RBIpioKEhHgsWmSHxXK2x8+ALoXIl7jrLiNmzvzJVf9c8ng8KCwsvGLwfH28Ho8Hr776Krze7o2V4UMkmaIouOuuVPzxjw8hPv5zeL0982gbITQoSj2ys4ciL28O+vfvf9X6mpoafPDBB9d84J+qqjhw4MBlD/70F8OHqA9QFAWDBsXhhRfm4Kc/DYWqfn7D0/BCCHR2nkdiohO///00zJhx53VdIO7o6Ljusxl/ar+N3+Gzf/9+3HvvvYiPj4eiKNi2bZtPuxACK1asQFxcHMLCwpCWloaamhqfmqamJmRlZcFsNiMqKgrz589Ha2trtzaEKNgpioKoqCg8/vjPsGLFJAwb1gCvtw6dnReu63qQpnWgo+Nz9O//CR5+2IwXXpiPkSOHX/fMVGRkJPr163fdtWFh3Zuh83uq/fz58xgzZgweffRRzJw587L2NWvWYN26ddi0aROSkpLw9NNPIz09HSdPntQ3LCsrC/X19SguLkZHRwfmzZuH7OxsbN68uVsbQ/RdoKoqJk0ah5SUkfjkk1ocOPA+Dh78BHV1HfB6IyHE1wNCg8HwJSIjgbFjo3HnneMxZsxwWCwWv5/JnpSUhJ/85Cd46623rvpaTdNw3333ISYm5ga38JJuPTRQURRs3boVM2bMAHDprCc+Ph5PPvkknnrqKQCAy+WC1WpFYWEhZs2ahaqqKiQnJ+Po0aOYMGECAGD37t2455578OmnnyI+Pv6a/fKhgXQzEUKgvb0dzc3N+PjjWjidX30tIyTEiLFjRyAyMuKGAuebTp48iezsbFy4cOGK7yWEwMCBA1FYWHjFY9WfY7NHP2R45swZOJ1OpKWl6essFgtSU1PhcDgwa9YsOBwOREVF6cEDAGlpaVBVFaWlpXjggQcue1+Px+Nzr1m3292Twybq0xRFgclkgtVq7fVHio8YMQL/+q//ipUrV8LlckFRFCiKAiEENE2DzWZDfn4+4uLiut1Xj4aP0+kEgMt2kNVq1ducTidiY2N9B2E0Ijo6Wq/5pvz8fKxcubInh0pEV3DpS6934vvf/z62bNmCXbt2oa2tDREREXjggQeQmZmJ2NjYbp9hAUHy9Yq8vDwsXbpU/9ntdiMhIUHiiIi+uxRFQWJiIpYtW4YFCxbA6/UiJCQEZrO5R0KnS4+Gj81mAwA0NDT4nJY1NDRg7Nixek1jY6PP6zo7O9HU1KS//ptMJhNMJlNPDpWIrkFRFAwYMKDX3r9HP+eTlJQEm82GkpISfZ3b7UZpaSnsdjsAwG63o7m5GWVlZXrN3r17oWkaUlNTe3I4RNSH+X3m09railOnTuk/nzlzBuXl5YiOjkZiYiIWL16MZ599FsOGDdOn2uPj4/UZsREjRuDuu+/GggULsGHDBnR0dGDhwoWYNWvWdc10EdF3hPDT22+/LQBctsydO1cIIYSmaeLpp58WVqtVmEwmMXXqVFFdXe3zHufOnROzZ88WERERwmw2i3nz5omWlpbrHoPL5RIAhMvl8nf4RNSL/Dk2u/U5H1n4OR+ivsmfY5Pf7SIiKRg+RCQFw4eIpGD4EJEUDB8ikoLhQ0RSMHyISAqGDxFJwfAhIikYPkQkBcOHiKRg+BCRFAwfIpKC4UNEUjB8iEgKhg8RScHwISIpGD5EJAXDh4ikYPgQkRQMHyKSguFDRFL4FT75+fmYOHEiIiMjERsbixkzZqC6utqnpq2tDbm5uYiJiUFERAQyMzPR0NDgU1NbW4uMjAyEh4cjNjYWy5YtQ2dnZ/e3hoiChl/hs2/fPuTm5uLw4cMoLi5GR0cHpk2bhvPnz+s1S5YswZtvvomioiLs27cPZ8+excyZM/V2r9eLjIwMtLe349ChQ9i0aRMKCwuxYsWKntsqIur7uvN0wsbGRgFA7Nu3TwghRHNzswgJCRFFRUV6TVVVlQAgHA6HEEKInTt3ClVVhdPp1GsKCgqE2WwWHo/nuvrlE0uJ+iZ/js1uXfNxuVwAgOjoaABAWVkZOjo6kJaWptcMHz4ciYmJcDgcAACHw4FRo0bBarXqNenp6XC73aisrLxiPx6PB26322chouB2w+GjaRoWL16MyZMnY+TIkQAAp9OJ0NBQREVF+dRarVY4nU695uvB09Xe1XYl+fn5sFgs+pKQkHCjwyaiPuKGwyc3NxcnTpzAli1benI8V5SXlweXy6UvdXV1vd4nEfUu4428aOHChdixYwf279+PQYMG6ettNhva29vR3Nzsc/bT0NAAm82m1xw5csTn/bpmw7pqvslkMsFkMt3IUImoj/LrzEcIgYULF2Lr1q3Yu3cvkpKSfNpTUlIQEhKCkpISfV11dTVqa2tht9sBAHa7HRUVFWhsbNRriouLYTabkZyc3J1tIaIg4teZT25uLjZv3ow33ngDkZGR+jUai8WCsLAwWCwWzJ8/H0uXLkV0dDTMZjMWLVoEu92OSZMmAQCmTZuG5ORkzJkzB2vWrIHT6cTy5cuRm5vLsxuim4k/02gArrhs3LhRr7l48aJ47LHHxIABA0R4eLh44IEHRH19vc/7fPLJJ2L69OkiLCxMDBw4UDz55JOio6PjusfBqXaivsmfY1MRQgh50Xdj3G43LBYLXC4XzGaz7OEQ0f/y59jkd7uISAqGDxFJwfAhIikYPkQkBcOHiKRg+BCRFAwfIpKC4UNEUjB8iEgKhg8RScHwISIpGD5EJAXDh4ikYPgQkRQMHyKSguFDRFIwfIhICoYPEUnB8CEiKRg+RCQFw4eIpGD4EJEUfoVPQUEBRo8eDbPZDLPZDLvdjl27duntbW1tyM3NRUxMDCIiIpCZmak/CrlLbW0tMjIyEB4ejtjYWCxbtgydnZ09szVEFDT8Cp9BgwZh9erVKCsrw7Fjx3DXXXfh/vvvR2VlJQBgyZIlePPNN1FUVIR9+/bh7NmzmDlzpv56r9eLjIwMtLe349ChQ9i0aRMKCwuxYsWKnt0qIur7uvuEwgEDBohXXnlFNDc3i5CQEFFUVKS3VVVVCQDC4XAIIYTYuXOnUFVVOJ1OvaagoECYzWbh8Xiuu08+sTR4aZrms9B3iz/H5g1f8/F6vdiyZQvOnz8Pu92OsrIydHR0IC0tTa8ZPnw4EhMT4XA4AAAOhwOjRo2C1WrVa9LT0+F2u/WzpyvxeDxwu90+CwUXIQTOnz+Pbdu24bHHHsM///M/4/nnn0dtbS1E8D00l3qA0d8XVFRUwG63o62tDREREdi6dSuSk5NRXl6O0NBQREVF+dRbrVY4nU4AgNPp9Amervautm+Tn5+PlStX+jtU6iOEEDhy5AjWrFmD06dPQ1EUAMDRo0exY8cOPPTQQ8jOzkZoaKjkkVIg+X3m88Mf/hDl5eUoLS1FTk4O5s6di5MnT/bG2HR5eXlwuVz6UldX16v9Uc8RQqC8vBy//vWvcebMGaiqCkVRoCgKVFXF+fPnsXHjRqxfvx6apskeLgWQ3+ETGhqKoUOHIiUlBfn5+RgzZgxefPFF2Gw2tLe3o7m52ae+oaEBNpsNAGCz2S6b/er6uavmSkwmkz7D1rVQcOjs7MSGDRvgcrn0M54ref3111FTUxPAkZFs3f6cj6Zp8Hg8SElJQUhICEpKSvS26upq1NbWwm63AwDsdjsqKirQ2Nio1xQXF8NsNiM5Obm7Q6E+6NixYygrK7tq8ACXPqbxH//xHzz7uYn4dc0nLy8P06dPR2JiIlpaWrB582a888472LNnDywWC+bPn4+lS5ciOjoaZrMZixYtgt1ux6RJkwAA06ZNQ3JyMubMmYM1a9bA6XRi+fLlyM3Nhclk6pUNJLlOnDiBzs5OGAyGq9apqoqKigp4vV6oKj/7ejPwK3waGxvxyCOPoL6+HhaLBaNHj8aePXvwD//wDwCAtWvXQlVVZGZmwuPxID09HevXr9dfbzAYsGPHDuTk5MBut6N///6YO3cuVq1a1bNbRX3Gtc54unTNeF1vPQU/RQThPKfb7YbFYoHL5eL1nz6uvLwc2dnZ8Hq9V63TNA0PPfQQfvWrX/HMJ4j5c2zyf5l61ahRozB58uSrfpZHCIGIiAg8/PDDDJ6bCP+nqVcZDAbk5OTAZrNdMYCEEDAYDPinf/onJCQkSBghycLwoV43bNgw/OEPf0BKSgqAS4EjhIDX64XVasVTTz2FrKwsnvXcZHjNhwKmvb0dR48exd///nd4vV4MHToUP/3pTzFgwABeaP6O8OfYZPgQUY/hBWci6vMYPkQkBcOHiKRg+BCRFAwfIpKC4UNEUjB8iEgKhg8RScHwISIpGD5EJAXDh4ikYPgQkRQMHyKSguFDRFIwfIhICoYPEUnB8CEiKboVPqtXr4aiKFi8eLG+rq2tDbm5uYiJiUFERAQyMzMve0RybW0tMjIyEB4ejtjYWCxbtgydnZ3dGQoRBZkbDp+jR4/i3/7t3zB69Gif9UuWLMGbb76JoqIi7Nu3D2fPnsXMmTP1dq/Xi4yMDLS3t+PQoUPYtGkTCgsLsWLFihvfCiIKPuIGtLS0iGHDhoni4mJxxx13iCeeeEIIIURzc7MICQkRRUVFem1VVZUAIBwOhxBCiJ07dwpVVYXT6dRrCgoKhNlsFh6P57r6d7lcAoBwuVw3Mnwi6iX+HJs3dOaTm5uLjIwMpKWl+awvKytDR0eHz/rhw4cjMTERDocDAOBwODBq1ChYrVa9Jj09HW63G5WVlVfsz+PxwO12+yxEFNz8elY7AGzZsgXHjx/H0aNHL2tzOp0IDQ1FVFSUz3qr1Qqn06nXfD14utq72q4kPz8fK1eu9HeoRNSH+XXmU1dXhyeeeAKvvfYa+vXr11tjukxeXh5cLpe+1NXVBaxvIuodfoVPWVkZGhsbMX78eBiNRhiNRuzbtw/r1q2D0WiE1WpFe3s7mpubfV7X0NAAm80GALDZbJfNfnX93FXzTSaTCWaz2WchouDmV/hMnToVFRUVKC8v15cJEyYgKytL/3dISAhKSkr011RXV6O2thZ2ux0AYLfbUVFRgcbGRr2muLgYZrMZycnJPbRZRNTX+XXNJzIyEiNHjvRZ179/f8TExOjr58+fj6VLlyI6OhpmsxmLFi2C3W7HpEmTAADTpk1DcnIy5syZgzVr1sDpdGL58uXIzc2FyWTqoc0ior7O7wvO17J27VqoqorMzEx4PB6kp6dj/fr1ervBYMCOHTuQk5MDu92O/v37Y+7cuVi1alVPD4WI+jA+q52Iegyf1U5EfR7Dh4ikYPgQkRQMHyKSguFDRFIwfIhICoYPEUnB8CEiKRg+RCQFw4eIpGD4EJEUDB8ikoLhQ0RSMHyISAqGDxFJwfAhIikYPkQkBcOHiKRg+BCRFAwfIpKC4UNEUjB8iEgKv8LnN7/5DRRF8VmGDx+ut7e1tSE3NxcxMTGIiIhAZmbmZY9Grq2tRUZGBsLDwxEbG4tly5ahs7OzZ7aGiIKG3w8N/NGPfoS///3vX72B8au3WLJkCf72t7+hqKgIFosFCxcuxMyZM3Hw4EEAgNfrRUZGBmw2Gw4dOoT6+no88sgjCAkJwXPPPdcDm0NEQUP44ZlnnhFjxoy5Yltzc7MICQkRRUVF+rqqqioBQDgcDiGEEDt37hSqqgqn06nXFBQUCLPZLDwez3WPw+VyCQDC5XL5M3wi6mX+HJt+X/OpqalBfHw8br31VmRlZaG2thYAUFZWho6ODqSlpem1w4cPR2JiIhwOBwDA4XBg1KhRsFqtek16ejrcbjcqKyu/tU+PxwO32+2zEFFw8yt8UlNTUVhYiN27d6OgoABnzpzB7bffjpaWFjidToSGhiIqKsrnNVarFU6nEwDgdDp9gqervavt2+Tn58NisehLQkKCP8Mmoj7Ir2s+06dP1/89evRopKamYvDgwfjv//5vhIWF9fjguuTl5WHp0qX6z263mwFEFOS6NdUeFRWFH/zgBzh16hRsNhva29vR3NzsU9PQ0ACbzQYAsNlsl81+df3cVXMlJpMJZrPZZyGi4Nat8GltbcXHH3+MuLg4pKSkICQkBCUlJXp7dXU1amtrYbfbAQB2ux0VFRVobGzUa4qLi2E2m5GcnNydoRBRkPHrz66nnnoK9957LwYPHoyzZ8/imWeegcFgwOzZs2GxWDB//nwsXboU0dHRMJvNWLRoEex2OyZNmgQAmDZtGpKTkzFnzhysWbMGTqcTy5cvR25uLkwmU69sIBH1TX6Fz6efforZs2fj3LlzuOWWWzBlyhQcPnwYt9xyCwBg7dq1UFUVmZmZ8Hg8SE9Px/r16/XXGwwG7NixAzk5ObDb7ejfvz/mzp2LVatW9exWEVGfpwghhOxB+MvtdsNiscDlcvH6D1Ef4s+xye92EZEUDB8ikoLhQ0RSMHyISAqGDxFJwfAhIin8vp8PUW/SNA2dnZ34/PMv0NTUrK83GFQMGZKI0NBQGAwGKIoib5DUIxg+JJ2maWhtbUVFxYd4++0KVFR8jpYWIy5cMOg1igLExHgQE2PC7bd/Hz/+8WgkJg5iEAUxfsiQpBFC4LPP6lFU9DYOHPgUX34ZAaMxGory7YEihAav9wIMhiYkJal46KHbcPvtExEaGhrg0dOV+HNsMnxIio6ODrz11iEUFpbjyy+joaomv89gLgVRE37841Dk5PwUcXGxPAuSzJ9jk392UcC1tLTi+eeLcPhwB4A4GAw3FhiKosJoHIjDhztQWfkaliy5A1OmjGcABQmGDwWMEAI1NWfw0kt7UFUVCVWN6JH3VdUQtLYOwpo1R3D69GeYPftu/hkWBDjVTgHT0tKK3/52Oz78cABUtWdvoaIoCjyeGLz6qhO7dh1AEF5NuOkwfCggLly4gN/97v/h7NmBUBTDtV9wAxRFgcEQhVde+QgOx3u90gf1HIYP9TohBLZv34/Dhzt6/Iznm7rOgNavP4DGxi96tS/qHoYP9bqamjPYvLkaBoMlIP0pioL6+gEoLNzDp+H2YQwf6lVerxd//etbuHgxsNPgBkM/vPXWF6isrA5Yn+Qfhg/1qv/5n1qUlZ2HqgZ2YlVRFChKLLZtO8SLz30Uw4d6jRAC7777PjQtRkr/imLAe+81we1ukdI/XR3Dh3qN1+vFgQOnYTCES+lfURS4XJH44IOTUvqnq2P4UK+pr6/HJ594pX7i2GgcgP37K6X1T9+O4UO9pqGhCV6v3OexKYoBDQ3nOevVB/kdPp999hl+/vOfIyYmBmFhYRg1ahSOHTumtwshsGLFCsTFxSEsLAxpaWmoqanxeY+mpiZkZWXBbDYjKioK8+fPR2tra/e3hvqUysqPoShRsoeB06dbcfHiRdnDoG/wK3y+/PJLTJ48GSEhIdi1axdOnjyJP/zhDxgwYIBes2bNGqxbtw4bNmxAaWkp+vfvj/T0dLS1tek1WVlZqKysRHFxMXbs2IH9+/cjOzu757aK6H/xS6Z9l1/zn7/73e+QkJCAjRs36uuSkpL0fwsh8MILL2D58uW4//77AQCvvvoqrFYrtm3bhlmzZqGqqgq7d+/G0aNHMWHCBADAn/70J9xzzz34/e9/j/j4+J7YLiLq4/w689m+fTsmTJiABx98ELGxsRg3bhxefvllvf3MmTNwOp1IS0vT11ksFqSmpsLhcAAAHA4HoqKi9OABgLS0NKiqitLS0iv26/F44Ha7fRYiCm5+hc/p06dRUFCAYcOGYc+ePcjJycHjjz+OTZs2AQCcTicAwGq1+rzOarXqbU6nE7GxsT7tRqMR0dHRes035efnw2Kx6EtCQoI/wyZJkpNvhRAuqWMQQuDWWyMQFhYmdRx0Ob/CR9M0jB8/Hs899xzGjRuH7OxsLFiwABs2bOit8QEA8vLy4HK59KWurq5X+6OeYbXGQFXbrl3Yi4TQEBsbDqORt67qa/wKn7i4OCQnJ/usGzFiBGprawEANpsNANDQ0OBT09DQoLfZbDY0Njb6tHd2dqKpqUmv+SaTyQSz2eyzUN8XHx+HIUNUqV9v6Oxswu23/0ha//Tt/AqfyZMno7ra94t6H330EQYPHgzg0sVnm82GkpISvd3tdqO0tBR2ux0AYLfb0dzcjLKyMr1m79690DQNqampN7wh1PcYDAbcfvut8HovSOlfCAGzuQVjxyZfu5gCzq/wWbJkCQ4fPoznnnsOp06dwubNm/GXv/wFubm5AC5Nay5evBjPPvsstm/fjoqKCjzyyCOIj4/HjBkzAFw6U7r77ruxYMECHDlyBAcPHsTChQsxa9YsznR9xyiKgilTxkJVz0npXwgN48YN4JlyH+XXH8ITJ07E1q1bkZeXh1WrViEpKQkvvPACsrKy9Jpf/vKXOH/+PLKzs9Hc3IwpU6Zg9+7d6Nevn17z2muvYeHChZg6dSpUVUVmZibWrVvXc1tFfcbgwQkYNy4c773XGdBvtgshIEQj7r//J/ysTx/FR+dQr/vww4/xy1/uQFtbXMCCwOttQ1qagqeemsWLzQHkz7HJ73ZRr/vhD2/FrFnD4PUG5vNZQgjExjZh3rx0Bk8fxvChXqcoCu6//05MnKhC09p7tS8hBEJDzyEnZzKs1lt6tS/qHoYPBUT//uHIy/u/sNkaIYTWK30IIeD1NuPRR4diypSUXumDeg7DhwLGbI7Ev/zLvRg69IsePwPqOuN5+OFbkJHxf3iROQgwfChgFEXBiBFDsXr1I5g4sQ1CNPfIBxA1rRPh4Z/iqacm4NFHZ8BkknsPIbo+DB8KOLM5Er/5zRzk5n4fkZH18Ho9NxRCQgh0dp7DxImteOmlWbjjjgk84wkinAogKUJDQ3HffT/BuHE/wOuvv4133z0DtzsSRuMAKIrhW0NECA1e70WoahMGDwYefHAC7rwzlWc7QYif8yHpNE2Dy+XG++9X4Z13TuDEiS/Q2hqCixdD9BpFERgwoA0xMaGYPHkIpkwZg8GDExASEsKznT7En2OT4UN9iqZp6OjoQENDI5qamvX1qqri1luHwGQKhdFoZOD0Uf4cm/yzi/oUVVVhMpmQmJiAxETet+m7jBeciUgKhg8RScHwISIpGD5EJAXDh4ikYPgQkRQMHyKSguFDRFIwfIhICoYPEUnB8CEiKRg+RCQFw4eIpGD4EJEUDB8ikiIo7+fTdf8ztzswD6EjouvTdUxezz0KgzJ8zp07BwBISODNpoj6opaWFlgslqvWBGX4REdHAwBqa2uvuYHfZW63GwkJCairq7upbyfL/XBJX9gPQgi0tLQgPj7+mrVBGT6qeulSlcViual/2bqYzWbuB3A/dJG9H673hIAXnIlICoYPEUkRlOFjMpnwzDPP3PQPiuN+uIT74ZJg2w9B+dwuIgp+QXnmQ0TBj+FDRFIwfIhICoYPEUkRlOHz5z//GUOGDEG/fv2QmpqKI0eOyB5Sj8nPz8fEiRMRGRmJ2NhYzJgxA9XV1T41bW1tyM3NRUxMDCIiIpCZmYmGhgafmtraWmRkZCA8PByxsbFYtmwZOjs7A7kpPWr16tVQFAWLFy/W190s++Gzzz7Dz3/+c8TExCAsLAyjRo3CsWPH9HYhBFasWIG4uDiEhYUhLS0NNTU1Pu/R1NSErKwsmM1mREVFYf78+WhtbQ30pvgSQWbLli0iNDRU/PWvfxWVlZViwYIFIioqSjQ0NMgeWo9IT08XGzduFCdOnBDl5eXinnvuEYmJiaK1tVWv+cUvfiESEhJESUmJOHbsmJg0aZL48Y9/rLd3dnaKkSNHirS0NPHee++JnTt3ioEDB4q8vDwZm9RtR44cEUOGDBGjR48WTzzxhL7+ZtgPTU1NYvDgweIf//EfRWlpqTh9+rTYs2ePOHXqlF6zevVqYbFYxLZt28T7778v7rvvPpGUlCQuXryo19x9991izJgx4vDhw+LAgQNi6NChYvbs2TI2SRd04XPbbbeJ3Nxc/Wev1yvi4+NFfn6+xFH1nsbGRgFA7Nu3TwghRHNzswgJCRFFRUV6TVVVlQAgHA6HEEKInTt3ClVVhdPp1GsKCgqE2WwWHo8nsBvQTS0tLWLYsGGiuLhY3HHHHXr43Cz74Ve/+pWYMmXKt7ZrmiZsNpt4/vnn9XXNzc3CZDKJ//qv/xJCCHHy5EkBQBw9elSv2bVrl1AURXz22We9N/hrCKo/u9rb21FWVoa0tDR9naqqSEtLg8PhkDiy3uNyuQB89WXasrIydHR0+OyD4cOHIzExUd8HDocDo0aNgtVq1WvS09PhdrtRWVkZwNF3X25uLjIyMny2F7h59sP27dsxYcIEPPjgg4iNjcW4cePw8ssv6+1nzpyB0+n02Q8WiwWpqak++yEqKgoTJkzQa9LS0qCqKkpLSwO3Md8QVOHzxRdfwOv1+vwyAYDVaoXT6ZQ0qt6jaRoWL16MyZMnY+TIkQAAp9OJ0NBQREVF+dR+fR84nc4r7qOutmCxZcsWHD9+HPn5+Ze13Sz74fTp0ygoKMCwYcOwZ88e5OTk4PHHH8emTZsAfLUdVzsmnE4nYmNjfdqNRiOio6Ol7oeg/Fb7zSI3NxcnTpzAu+++K3soAVdXV4cnnngCxcXF6Nevn+zhSKNpGiZMmIDnnnsOADBu3DicOHECGzZswNy5cyWPrnuC6sxn4MCBMBgMl81oNDQ0wGazSRpV71i4cCF27NiBt99+G4MGDdLX22w2tLe3o7m52af+6/vAZrNdcR91tQWDsrIyNDY2Yvz48TAajTAajdi3bx/WrVsHo9EIq9V6U+yHuLg4JCcn+6wbMWIEamtrAXy1HVc7Jmw2GxobG33aOzs70dTUJHU/BFX4hIaGIiUlBSUlJfo6TdNQUlICu90ucWQ9RwiBhQsXYuvWrdi7dy+SkpJ82lNSUhASEuKzD6qrq1FbW6vvA7vdjoqKCp9fuOLiYpjN5st+kfuqqVOnoqKiAuXl5foyYcIEZGVl6f++GfbD5MmTL/uoxUcffYTBgwcDAJKSkmCz2Xz2g9vtRmlpqc9+aG5uRllZmV6zd+9eaJqG1NTUAGzFt5B2qfsGbdmyRZhMJlFYWChOnjwpsrOzRVRUlM+MRjDLyckRFotFvPPOO6K+vl5fLly4oNf84he/EImJiWLv3r3i2LFjwm63C7vdrrd3TTFPmzZNlJeXi927d4tbbrklqKaYr+Trs11C3Bz74ciRI8JoNIrf/va3oqamRrz22msiPDxc/Od//qdes3r1ahEVFSXeeOMN8cEHH4j777//ilPt48aNE6WlpeLdd98Vw4YN41T7jfjTn/4kEhMTRWhoqLjtttvE4cOHZQ+pxwC44rJx40a95uLFi+Kxxx4TAwYMEOHh4eKBBx4Q9fX1Pu/zySefiOnTp4uwsDAxcOBA8eSTT4qOjo4Ab03P+mb43Cz74c033xQjR44UJpNJDB8+XPzlL3/xadc0TTz99NPCarUKk8kkpk6dKqqrq31qzp07J2bPni0iIiKE2WwW8+bNEy0tLYHcjMvwlhpEJEVQXfMhou8Ohg8RScHwISIpGD5EJAXDh4ikYPgQkRQMHyKSguFDRFIwfIhICoYPEUnB8CEiKRg+RCTF/wetzODdDebvcgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 300x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gym\n",
    "\n",
    "\n",
    "#定义环境\n",
    "class MyWrapper(gym.Wrapper):\n",
    "\n",
    "    def __init__(self):\n",
    "        from pettingzoo.mpe import simple_spread_v3\n",
    "        self.N = 2\n",
    "        env = simple_spread_v3.env(N=self.N,\n",
    "                                   local_ratio=0.5,\n",
    "                                   max_cycles=1e8,\n",
    "                                   render_mode='rgb_array')\n",
    "        super().__init__(env)\n",
    "        self.env = env\n",
    "        self.step_n = 0\n",
    "\n",
    "    def reset(self):\n",
    "        self.env.reset()\n",
    "        self.step_n = 0\n",
    "\n",
    "        #不允许两个目标点靠得太近\n",
    "        import numpy as np\n",
    "        mark0, mark1 = self.env.env.env.world.landmarks\n",
    "        dist = np.array(mark0.state.p_pos) - np.array(mark1.state.p_pos)\n",
    "        dist = (dist**2).sum()**0.5\n",
    "        if dist < 1:\n",
    "            return self.reset()\n",
    "\n",
    "        return self.state()\n",
    "\n",
    "    def state(self):\n",
    "        state = []\n",
    "        for i in self.env.agents:\n",
    "            state.append(env.observe(i).tolist())\n",
    "        return state\n",
    "\n",
    "    def step(self, action):\n",
    "        #走一步停N步,取消惯性.\n",
    "        reward_sum = [0] * self.N\n",
    "        for i in range(5):\n",
    "            if i != 0:\n",
    "                action = [-1, -1]\n",
    "            next_state, reward, over = self._step(action)\n",
    "            for j in range(self.N):\n",
    "                reward_sum[j] += reward[j]\n",
    "            self.step_n -= 1\n",
    "\n",
    "        self.step_n += 1\n",
    "\n",
    "        return next_state, reward_sum, over\n",
    "\n",
    "    def _step(self, action):\n",
    "        for i, _ in enumerate(env.agent_iter(self.N)):\n",
    "            self.env.step(action[i] + 1)\n",
    "\n",
    "        reward = [self.env.rewards[i] for i in self.env.agents]\n",
    "\n",
    "        _, _, termination, truncation, _ = env.last()\n",
    "        over = termination or truncation\n",
    "\n",
    "        #限制最大步数\n",
    "        self.step_n += 1\n",
    "        if self.step_n >= 50:\n",
    "            over = True\n",
    "\n",
    "        return self.state(), reward, over\n",
    "\n",
    "    #打印游戏图像\n",
    "    def show(self):\n",
    "        from matplotlib import pyplot as plt\n",
    "        plt.figure(figsize=(3, 3))\n",
    "        plt.imshow(self.env.render())\n",
    "        plt.show()\n",
    "\n",
    "\n",
    "env = MyWrapper()\n",
    "env.reset()\n",
    "\n",
    "env.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<__main__.A2C at 0x7f8951fe8a00>, <__main__.A2C at 0x7f88df748be0>]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "class A2C:\n",
    "\n",
    "    def __init__(self, model_actor, model_critic, model_critic_delay,\n",
    "                 optimizer_actor, optimizer_critic):\n",
    "        self.model_actor = model_actor\n",
    "        self.model_critic = model_critic\n",
    "        self.model_critic_delay = model_critic_delay\n",
    "        self.optimizer_actor = optimizer_actor\n",
    "        self.optimizer_critic = optimizer_critic\n",
    "\n",
    "        self.model_critic_delay.load_state_dict(self.model_critic.state_dict())\n",
    "        self.requires_grad(self.model_critic_delay, False)\n",
    "\n",
    "    def soft_update(self, _from, _to):\n",
    "        for _from, _to in zip(_from.parameters(), _to.parameters()):\n",
    "            value = _to.data * 0.99 + _from.data * 0.01\n",
    "            _to.data.copy_(value)\n",
    "\n",
    "    def requires_grad(self, model, value):\n",
    "        for param in model.parameters():\n",
    "            param.requires_grad_(value)\n",
    "\n",
    "    def train_critic(self, state, reward, next_state, over):\n",
    "        self.requires_grad(self.model_critic, True)\n",
    "        self.requires_grad(self.model_actor, False)\n",
    "\n",
    "        #计算values和targets\n",
    "        value = self.model_critic(state)\n",
    "\n",
    "        with torch.no_grad():\n",
    "            target = self.model_critic_delay(next_state)\n",
    "        target = target * 0.99 * (1 - over) + reward\n",
    "\n",
    "        #时序差分误差,也就是tdloss\n",
    "        loss = torch.nn.functional.mse_loss(value, target)\n",
    "\n",
    "        loss.backward()\n",
    "        self.optimizer_critic.step()\n",
    "        self.optimizer_critic.zero_grad()\n",
    "        self.soft_update(self.model_critic, self.model_critic_delay)\n",
    "\n",
    "        #减去value相当于去基线\n",
    "        return (target - value).detach()\n",
    "\n",
    "    def train_actor(self, state, action, value):\n",
    "        self.requires_grad(self.model_critic, False)\n",
    "        self.requires_grad(self.model_actor, True)\n",
    "\n",
    "        #重新计算动作的概率\n",
    "        prob = self.model_actor(state)\n",
    "        prob = prob.gather(dim=1, index=action)\n",
    "\n",
    "        #根据策略梯度算法的导函数实现\n",
    "        #函数中的Q(state,action),这里使用critic模型估算\n",
    "        prob = (prob + 1e-8).log() * value\n",
    "        loss = -prob.mean()\n",
    "\n",
    "        loss.backward()\n",
    "        self.optimizer_actor.step()\n",
    "        self.optimizer_actor.zero_grad()\n",
    "\n",
    "        return loss.item()\n",
    "\n",
    "\n",
    "model_actor = [\n",
    "    torch.nn.Sequential(\n",
    "        torch.nn.Linear(6 * env.N * env.N, 64),\n",
    "        torch.nn.ReLU(),\n",
    "        torch.nn.Linear(64, 64),\n",
    "        torch.nn.ReLU(),\n",
    "        torch.nn.Linear(64, 4),\n",
    "        torch.nn.Softmax(dim=1),\n",
    "    ) for _ in range(env.N)\n",
    "]\n",
    "\n",
    "model_critic, model_critic_delay = [\n",
    "    torch.nn.Sequential(\n",
    "        torch.nn.Linear(6 * env.N * env.N, 64),\n",
    "        torch.nn.ReLU(),\n",
    "        torch.nn.Linear(64, 64),\n",
    "        torch.nn.ReLU(),\n",
    "        torch.nn.Linear(64, 1),\n",
    "    ) for _ in range(2)\n",
    "]\n",
    "\n",
    "optimizer_actor = [\n",
    "    torch.optim.Adam(model_actor[i].parameters(), lr=1e-3)\n",
    "    for i in range(env.N)\n",
    "]\n",
    "optimizer_critic = torch.optim.Adam(model_critic.parameters(), lr=5e-3)\n",
    "\n",
    "a2c = [\n",
    "    A2C(model_actor[i], model_critic, model_critic_delay, optimizer_actor[i],\n",
    "        optimizer_critic) for i in range(env.N)\n",
    "]\n",
    "\n",
    "model_actor = None\n",
    "model_critic = None\n",
    "model_critic_delay = None\n",
    "optimizer_actor = None\n",
    "optimizer_critic = None\n",
    "\n",
    "a2c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-514.2003784179688"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython import display\n",
    "import random\n",
    "\n",
    "\n",
    "#玩一局游戏并记录数据\n",
    "def play(show=False):\n",
    "    state = []\n",
    "    action = []\n",
    "    reward = []\n",
    "    next_state = []\n",
    "    over = []\n",
    "\n",
    "    s = env.reset()\n",
    "    o = False\n",
    "    while not o:\n",
    "        a = []\n",
    "        for i in range(env.N):\n",
    "            #计算动作\n",
    "            prob = a2c[i].model_actor(torch.FloatTensor(s).reshape(\n",
    "                1, -1))[0].tolist()\n",
    "            a.append(random.choices(range(4), weights=prob, k=1)[0])\n",
    "\n",
    "        #执行动作\n",
    "        ns, r, o = env.step(a)\n",
    "\n",
    "        state.append(s)\n",
    "        action.append(a)\n",
    "        reward.append(r)\n",
    "        next_state.append(ns)\n",
    "        over.append(o)\n",
    "\n",
    "        s = ns\n",
    "\n",
    "        if show:\n",
    "            display.clear_output(wait=True)\n",
    "            env.show()\n",
    "\n",
    "    state = torch.FloatTensor(state)\n",
    "    action = torch.LongTensor(action).unsqueeze(-1)\n",
    "    reward = torch.FloatTensor(reward).unsqueeze(-1)\n",
    "    next_state = torch.FloatTensor(next_state)\n",
    "    over = torch.LongTensor(over).reshape(-1, 1)\n",
    "\n",
    "    return state, action, reward, next_state, over, reward.sum().item()\n",
    "\n",
    "\n",
    "state, action, reward, next_state, over, reward_sum = play()\n",
    "\n",
    "reward_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 -27.129926681518555 -483.3706680297852\n",
      "2500 -1.9944206476211548 -171.64251594543458\n",
      "5000 0.2974016070365906 -124.18828353881835\n",
      "7500 0.22703175246715546 -128.68980560302734\n",
      "10000 0.6404007077217102 -108.4465202331543\n",
      "12500 0.001281398581340909 -102.88806457519532\n",
      "15000 -0.13608776032924652 -82.45703506469727\n",
      "17500 0.2289573848247528 -94.22195186614991\n",
      "20000 0.04599471017718315 -94.12368965148926\n",
      "22500 -0.23084907233715057 -94.51665592193604\n",
      "25000 -1.0701826810836792 -108.04108600616455\n",
      "27500 0.014983355067670345 -107.3333625793457\n",
      "30000 -0.053469013422727585 -81.96413516998291\n",
      "32500 0.11448055505752563 -89.71076755523681\n",
      "35000 -0.012496666982769966 -90.8791841506958\n",
      "37500 -0.09561604261398315 -85.39328899383545\n",
      "40000 -0.11113201081752777 -98.64780101776122\n",
      "42500 0.01348934043198824 -95.79078578948975\n",
      "45000 0.0001955045881913975 -80.28663330078125\n",
      "47500 0.0003153890138491988 -104.6043041229248\n"
     ]
    }
   ],
   "source": [
    "def train():\n",
    "    #训练N局\n",
    "    for epoch in range(5_0000):\n",
    "        state, action, reward, next_state, over, _ = play()\n",
    "\n",
    "        #合并部分字段\n",
    "        state_c = state.flatten(start_dim=1)\n",
    "        reward_c = reward.sum(dim=1)\n",
    "        next_state_c = next_state.flatten(start_dim=1)\n",
    "\n",
    "        for i in range(env.N):\n",
    "            value = a2c[i].train_critic(state_c, reward_c, next_state_c, over)\n",
    "            loss = a2c[i].train_actor(state_c, action[:, i], value)\n",
    "\n",
    "        if epoch % 2500 == 0:\n",
    "            test_result = sum([play()[-1] for _ in range(20)]) / 20\n",
    "            print(epoch, loss, test_result)\n",
    "\n",
    "\n",
    "train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAR8AAAEYCAYAAABlUvL1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmlklEQVR4nO3dfVRU950G8GfeeZ0ZQZnBCIoJLbK+FCHiRJNsIxFT8taYl3qMMa4bNywalawn5ZyoTboN1mw3jbuJbrOn0Z7GJKU9JtH6cihWrHFEJTEiJkRXDUQdUJAZUBmYme/+QbnJxDeGtwvyfM6558j9fe/c371wH+/cV42ICIiI+phW7Q4Q0eDE8CEiVTB8iEgVDB8iUgXDh4hUwfAhIlUwfIhIFQwfIlIFw4eIVMHwISJVqBY+b7zxBkaNGoWwsDBkZmZi//79anWFiFSgSvi8//77yM/Px8qVK/HJJ59gwoQJyM7ORl1dnRrdISIVaNS4sTQzMxO33347/vu//xsAEAgEkJCQgEWLFuGnP/1pX3eHiFSg7+sZtra2ory8HAUFBco4rVaLrKwsOJ3Oq07j9Xrh9XqVnwOBABoaGhAbGwuNRtPrfSaizhERNDU1Yfjw4dBqr//Fqs/D5/z58/D7/bDZbEHjbTYbvvjii6tOU1hYiJdeeqkvukdEPaCmpgYjRoy4bk2fh09XFBQUID8/X/nZ7XYjMTERNTU1MJvNKvaMiL7N4/EgISEB0dHRN6zt8/AZOnQodDodamtrg8bX1tbCbrdfdRqTyQSTyXTFeLPZzPAh6oc6czikz892GY1GpKeno6SkRBkXCARQUlICh8PR190hIpWo8rUrPz8fc+fORUZGBiZNmoRf//rXuHjxIubNm6dGd4hIBaqEzxNPPIFz585hxYoVcLlc+MEPfoDt27dfcRCaiG5eqlzn010ejwcWiwVut5vHfIj6kVC2Td7bRUSqYPgQkSoYPkSkCoYPEamC4UNEqmD4EJEqGD5EpAqGDxGpguFDRKpg+BCRKhg+RKQKhg8RqYLhQ0SqYPgQkSoYPkSkCoYPEamC4UNEqmD4EJEqGD5EpAqGDxGpguFDRKpg+BCRKkIOn927d+OBBx7A8OHDodFo8MEHHwS1iwhWrFiB+Ph4hIeHIysrC8eOHQuqaWhowOzZs2E2m2G1WjF//nw0Nzd3a0GIaGAJOXwuXryICRMm4I033rhq++rVq7FmzRqsW7cOZWVliIyMRHZ2NlpaWpSa2bNno7KyEsXFxdiyZQt2796NBQsWdH0piGjgkW4AIJs2bVJ+DgQCYrfb5dVXX1XGNTY2islkknfffVdERI4ePSoA5MCBA0rNtm3bRKPRyOnTpzs1X7fbLQDE7XZ3p/tE1MNC2TZ79JjPyZMn4XK5kJWVpYyzWCzIzMyE0+kEADidTlitVmRkZCg1WVlZ0Gq1KCsru+rner1eeDyeoIGIBrYeDR+XywUAV7xz3WazKW0ulwtxcXFB7Xq9HjExMUrNdxUWFsJisShDQkJCT3abiFQwIM52FRQUwO12K0NNTY3aXSKiburR8LHb7QCA2traoPG1tbVKm91uR11dXVC7z+dDQ0ODUvNdJpMJZrM5aCCiga1HwycpKQl2ux0lJSXKOI/Hg7KyMjgcDgCAw+FAY2MjysvLlZqdO3ciEAggMzOzJ7tDRP2YPtQJmpubcfz4ceXnkydP4tChQ4iJiUFiYiKWLFmCf//3f0dycjKSkpKwfPlyDB8+HA8//DAAYMyYMZgxYwaeeeYZrFu3Dm1tbVi4cCF+8pOfYPjw4T22YETUz4V6Ku2vf/2rALhimDt3roi0n25fvny52Gw2MZlMMm3aNKmqqgr6jPr6epk1a5ZERUWJ2WyWefPmSVNTU6f7wFPtRP1TKNumRkRExezrEo/HA4vFArfbzeM/RP1IKNvmgDjbRUQ3H4YPEamC4UNEqmD4EJEqQj7VTkQ3LxGB3+9HfX09LlxoVMZrNFokJo6A0WiEVquFRqPp9rwYPkSDnIjg4sVLOHq0Crt2HcahQy54PDo0N+uUGo0GGDasDbGxJtx55624447xGDHiFuh0ui4HEU+1Ew1SIoLa2jr86U+7sGvXVzh/Phx6fSw0Gv01A0UkAL//EvT6BiQl6fD445MwdWoGjEYjgNC2TYYP0SDk8/mwa1cZ/vd/D+L8+SHQasNC3oNpD6IGTJlixLPP5iA+3oampqZOb5v82kU0yFy6dAmvvfYn7N59ESLx0Om6+LVJo4VePxROZxuOHNmIpUvvxvjxt3V6eoYP0SAhIjh1qgZvvLENn30WAa02Fj1w3BharQHNzSOwevV+zJjxeaenY/gQDRKXLl3GK698gFOn4qDV6m48QQg0Gg283lj84Q/1nZ6G1/kQDQItLS341a/+hFOnhvR48HTQaDTQ6yM6Xc/wIbrJiQi2b/8Yu3dfhlYbpnZ3FAwfopvcV199jQ0bKqDVWtXuShCGD9FNLBAI4O23t6O5Oa5HrkruSQwfopvY6dNnsG9fI7Rag9pduQLDh+gmJSJwOj+DzxerdleuiuFDdJMKBAIoLT0GnS5S7a5cFcOH6CZVV1eH48dbodH0z82cFxkS3aTOnbuAtjYjDF043CPix8WLNXC7yxEIeKHTRcJqvR3h4fYeCzOGD9FNqrLyOABryNO1trpRV7cVwAmYTO1BIyKorz8KnS4FcXHZIV1MeC39c3+MiFTR1taMc+eKYDSegsnU/tAwjUYDrVaLsDBAr/8cdXUfwO/3dnteIYVPYWEhbr/9dkRHRyMuLg4PP/wwqqqqgmpaWlqQl5eH2NhYREVFYebMmVe8Prm6uho5OTmIiIhAXFwcli1bBp/P1+2FIaKuExFcuHAQOl0ttFrNFdcFdYQQcAJu91F092k8IYVPaWkp8vLysG/fPhQXF6OtrQ3Tp0/HxYsXlZqlS5di8+bNKCoqQmlpKc6cOYNHHnlEaff7/cjJyUFrayv27t2LDRs2YP369VixYkW3FoSIusfna0ZLyyfQ6a4fCwaDDs3N+xAItHZrft16mNi5c+cQFxeH0tJS3HXXXXC73Rg2bBg2btyIRx99FADwxRdfYMyYMXA6nZg8eTK2bduG+++/H2fOnIHNZgMArFu3Di+88ALOnTunPBHtevryYWIiAhFBQ0MDAoGAMt5kMinz7m9XjhIBwKefVuD553fCYOjca8ibmk7B7f4dwsJuvA1euuRDXFwuwsKCryHy+S5h586ne/9hYm63GwAQExMDACgvL0dbWxuysrKUmpSUFCQmJirh43Q6MW7cOCV4ACA7Oxu5ubmorKxEWlraFfPxer3wer/5junxeLrT7RsSETQ1NeHIkS9RWlqBmpoLOHmyDT7fNzltNmsxfLgB48aNwJ13TsDo0SNhNBoZRNRv2Gyx0Ou7f2ymt3Q5fAKBAJYsWYIpU6Zg7NixAACXywWj0Qir1RpUa7PZ4HK5lJpvB09He0fb1RQWFuKll17qalc7LRAI4MSJUygqKkV5+Xk0NERAq7VCqx3x9++739Q2NQmqqoCjRy/h/ff/jIQEP374w2Tcf/+diIkZwhAi1dlsNowebcCJE9Kpv8ewsFhcuBAFEe9169u/LMXCYIjqVv+6HD55eXk4cuQI9uzZ060OdEZBQQHy8/OVnz0eDxISEnrs80UELS1e/PGPxdi48Tja2uKg04267vURHb+c9lOOEThzRvC73zVg27b/xf33j4DBoEN4eDimTp2KIUMYRtT3tFot7rrrNhw7dh56/Y2DQq+PQljYRPj9H0Ovv/Yzf9ra/IiKmgyt9sZfz647v65MtHDhQmzZsgW7d+/GiBEjlPF2ux2tra1obGwM2vupra2F3W5Xavbv3x/0eR1nwzpqvstkMsFkMnWlqzckIqiuPoPVqzfhiy8ioNPd0sVn2gbg8XyJ06f34siReuWVIvHx8fjJT36Cxx9/HOHh4T3ef6Jr0Wg0mDJlAtav/wOAG4ePRqPBkCEZqKs7Dq32HDSa4OOZHcc/gSRYLKnd/g81pLNdIoKFCxdi06ZN2LlzJ5KSkoLa09PTYTAYUFJSooyrqqpCdXU1HA4HAMDhcKCiogJ1dXVKTXFxMcxmM1JTU7uzLCFrv/HuEyxf/iGOHRsKvT66SytUxI/6+r/B692OiIgmmEwm6PV66HQ61NbW4te//jV++ctf4vLly72wFETXNmLELUhPj0Yg0LlLWQyGKAwb9hhaW0fC6w0ogRMIBNDSAvh8KYiL+zF0uu7vDIS055OXl4eNGzfiww8/RHR0tHKMxmKxIDw8HBaLBfPnz0d+fj5iYmJgNpuxaNEiOBwOTJ48GQAwffp0pKamYs6cOVi9ejVcLhdefPFF5OXl9drezdWICA4f/hwvv7wHgUDXLxkXETQ3V6OlZS/Cwq78jI6LtD788EOkpaXhwQcf5Fcw6jM6nQ7/9E/34ujRzbh8Ob5Tf3tGowW33PLEFbdXxMb27O0VIZ1qv1bH3377bTz99NMA2i8yfP755/Huu+/C6/UiOzsbb775ZtBXqq+++gq5ubnYtWsXIiMjMXfuXKxatQp6feeysCdOtVdXn8by5R/h7Nkh0Gi6/kxbET9On34PJlP1DQ/SjR49GuvXr0dUVPcO1BGFQkTw/vs78NZbNdDrrb06r1BOtQ/Klwa2tLRgxYrf4dNPo6HVdu/2Nq+3ES7Xm4iMvPH/Bn6/H7/97W8xceLEbs2TKFSXLl3Cz3/+Pg4eNECr7b1vGKGEz6C7t0tEsG3bHpSXo9vB8/dPBBC4YRXQfir/2xcqEvWViIgIFBQ8iltuOQ+R3vkbFJGQ7vkadOFTW3sOb799CDqdtUc+r/3AW+QN73MREURERCoXZBL1tejoKPz0pw/gttvOIxDo2YsPRQR6fQN+9KPOH8IYVOHT/gqRPbh4cViPHfTV6cIRETERfv/1/zcREVy+PAQtLW09Ml+iUGk0GqSk3Ipf/nIuJk1qRSBwods3hwLtxz1Npq+xZMl4/Mu/PNrp6QZV+Ph8PuzZcwo6Xc+9u0ij0cBqTUdbm/Wav0gRQWurCVFRP8Tu3Z/1yC+cqKuio6OwcuWTWLjwNkRHn4Xf7+3S36SIwOe7gPHjG7FmzaO4915HSP+pD6rwOXWqGl99hR5/rKReH4lhwx5HS4sVbW0+5RfZ/svx4/LlcFitP0Zk5Eg4ndVobe3e3cBE3WU0GvHggz/Ef/7nI7j33gDCwk6itfU8RPzXDaL24zqX4fd/jfj4aixefCt+/vOnkZSUGPK3iUH1JMOPP/4MgUBs0D1aPUGj0cBkGor4+HnweI6gqWk/RC5DozEgIuIHiI+fCL0+EhqNBl9/rcOJE6cwZsz3e7YTRCHSaDQYOTIBy5Y9iQUL3Pj006PYtesIjhxpwKVLRly+bPhWLWCxXMKQIXo4HIm4664fYfToUd26mXrQhI+I4OzZxl57XWz7e6rDMGRIOqzWH0Ak8PcLDPVBvxy/34hz5y5gzJhe6QZRyLRaLYYMGYJ77pmCf/xHB1pbW1FbW4dz5+qDam67LQlhYWEwGAw9csx00ISPz+fD4cN10GhG9+p8OgLnWgyGIThwoAp33TW5V/tB1BXtj0sNw8iRiRg5MrF359Wrn97PBAKde7RA79LwWh8iDLLwIaL+g+FDRKoYNOGj1WoxcqQFgYBf1X74/Rdx662de6Yu0c1s0ISPTqdDYqIFIuqGTyBwEUlJ8ar2gag/GDThAwCZmanw+xtUm7+IICamBbfeOkq1PhD1F4MqfFJTv4ehQy+rdnuDiB8TJw7l83yIMMjCJyIiApMm2SCizttRfb4G/PCH4/7+1keiwW1QbQUajQYzZkyCRnOuz+ctIhgxwosJE/r2OdVE/dWgCh+g/avXPffEwu/v64e5N2DOnExERkb28XyJ+qdBFz7tD9TOhs3W2GfHfvz+Ftx5ZzjuuSezH1xhTdQ/DLrwAYBhw4ZiwYLJ0Osbej2ARAKw293453/O7vQD8okGg0EZPhqNBnfddTsWLEiG39/Ya/MRERiNZ/Czn90Pu9124wmIBpFBGT5Ax8HnKXjgATO02voe3wMSCSA6+izy8jIwenToD1oiutmFFD5r167F+PHjYTabYTab4XA4sG3bNqW9paUFeXl5iI2NRVRUFGbOnKm8CrlDdXU1cnJyEBERgbi4OCxbtgw+nzqnvsPCwvDcc49j6dLx0OtP99jVz4FAC2w2F/7jP36MGTOm8tQ60VWEtFWMGDECq1atQnl5OQ4ePIh77rkHDz30ECorKwEAS5cuxebNm1FUVITS0lKcOXMGjzzyiDK93+9HTk4OWltbsXfvXmzYsAHr16/HihUrenapQqDVanHvvQ68/vpD+N73zsPn83R5L6j9lSTnMH26Bq+//hRGjUrgHg/RNXT7pYExMTF49dVX8eijj2LYsGHYuHEjHn20/Qn2X3zxBcaMGQOn04nJkydj27ZtuP/++3HmzBnYbO3HQNatW4cXXngB586dg9Fo7NQ8e+KNpd8lImhp8WLTpr/ggw8+x/nz4dBqh0Crvf5T29qf03wZWm09kpN1mD37DtxxRzr3dmhQCmXb7PLpF7/fj6KiIly8eBEOhwPl5eVoa2tDVlaWUpOSkoLExEQlfJxOJ8aNG6cEDwBkZ2cjNzcXlZWVSEtLu+q8vF4vvN5v3jPk8Xi62u1r0mg0CA8Pw6xZOXjggbtRWfklSksrcPDgGTQ2BuDzWSHyTaBota3Q6TxISDBiypRRmDr1DowalditZ9oSDSYhh09FRQUcDgdaWloQFRWFTZs2ITU1FYcOHYLRaITVag2qt9lscLlcAACXyxUUPB3tHW3XUlhYiJdeeinUrnaJRqNBdHQ0Jk9OR2bmRDQ3N6O5+SIOH/4Cra3fvHMrNtaK5ORRMJvNDByiLgg5fL7//e/j0KFDcLvd+OMf/4i5c+eitLS0N/qmKCgoQH5+vvKzx+NBQkJCr84T+CaIoqOjER9v7/X5EQ0mIYeP0WjEbbfdBgBIT0/HgQMH8Prrr+OJJ55Aa2srGhsbg/Z+amtrYbe3b7h2ux379+8P+ryOs2EdNVdjMplgMvXey+2JqO91+6hoIBCA1+tFeno6DAYDSkpKlLaqqipUV1fD4XAAABwOByoqKlBXV6fUFBcXw2w2IzWVN1wSDSYh7fkUFBTgvvvuQ2JiIpqamrBx40bs2rULO3bsgMViwfz585Gfn4+YmBiYzWYsWrQIDocDkye3vyZm+vTpSE1NxZw5c7B69Wq4XC68+OKLyMvL454N0SATUvjU1dXhqaeewtmzZ2GxWDB+/Hjs2LED9957LwDgtddeg1arxcyZM+H1epGdnY0333xTmV6n02HLli3Izc2Fw+FAZGQk5s6di5dffrlnl4qI+r1uX+ejht64zoeIui+UbZNXwhGRKhg+RKQKhg8RqYLhQ0SqYPgQkSoYPkSkCoYPEamC4UNEqmD4EJEqGD5EpAqGDxGpguFDRKpg+BCRKhg+RKQKhg8RqYLhQ0SqYPgQkSoYPkSkCoYPEamC4UNEqmD4EJEqGD5EpIpuhc+qVaug0WiwZMkSZVxLSwvy8vIQGxuLqKgozJw5U3klcofq6mrk5OQgIiICcXFxWLZsGXw+X3e6QkQDTJfD58CBA/if//kfjB8/Pmj80qVLsXnzZhQVFaG0tBRnzpzBI488orT7/X7k5OSgtbUVe/fuxYYNG7B+/XqsWLGi60tBRAOPdEFTU5MkJydLcXGx3H333bJ48WIREWlsbBSDwSBFRUVK7eeffy4AxOl0iojI1q1bRavVisvlUmrWrl0rZrNZvF5vp+bvdrsFgLjd7q50n4h6SSjbZpf2fPLy8pCTk4OsrKyg8eXl5Whrawsan5KSgsTERDidTgCA0+nEuHHjYLPZlJrs7Gx4PB5UVlZedX5erxcejydoIKKBLaR3tQPAe++9h08++QQHDhy4os3lcsFoNMJqtQaNt9lscLlcSs23g6ejvaPtagoLC/HSSy+F2lUi6sdC2vOpqanB4sWL8c477yAsLKy3+nSFgoICuN1uZaipqemzeRNR7wgpfMrLy1FXV4eJEydCr9dDr9ejtLQUa9asgV6vh81mQ2trKxobG4Omq62thd1uBwDY7fYrzn51/NxR810mkwlmszloIKKBLaTwmTZtGioqKnDo0CFlyMjIwOzZs5V/GwwGlJSUKNNUVVWhuroaDocDAOBwOFBRUYG6ujqlpri4GGazGampqT20WETU34V0zCc6Ohpjx44NGhcZGYnY2Fhl/Pz585Gfn4+YmBiYzWYsWrQIDocDkydPBgBMnz4dqampmDNnDlavXg2Xy4UXX3wReXl5MJlMPbRYRNTfhXzA+UZee+01aLVazJw5E16vF9nZ2XjzzTeVdp1Ohy1btiA3NxcOhwORkZGYO3cuXn755Z7uChH1YxoREbU7ESqPxwOLxQK3283jP0T9SCjbJu/tIiJVMHyISBUMHyJSBcOHiFTB8CEiVTB8iEgVDB8iUgXDh4hUwfAhIlUwfIhIFQwfIlIFw4eIVMHwISJVMHyISBUMHyJSBcOHiFTB8CEiVTB8iEgVDB8iUgXDh4hUwfAhIlUwfIhIFSGFz89+9jNoNJqgISUlRWlvaWlBXl4eYmNjERUVhZkzZ17xauTq6mrk5OQgIiICcXFxWLZsGXw+X88sDRENGCG/NPAf/uEf8Je//OWbD9B/8xFLly7Fn//8ZxQVFcFisWDhwoV45JFH8PHHHwMA/H4/cnJyYLfbsXfvXpw9exZPPfUUDAYDXnnllR5YHCIaMCQEK1eulAkTJly1rbGxUQwGgxQVFSnjPv/8cwEgTqdTRES2bt0qWq1WXC6XUrN27Voxm83i9Xo73Q+32y0AxO12h9J9IuploWybIR/zOXbsGIYPH47Ro0dj9uzZqK6uBgCUl5ejra0NWVlZSm1KSgoSExPhdDoBAE6nE+PGjYPNZlNqsrOz4fF4UFlZec15er1eeDyeoIGIBraQwiczMxPr16/H9u3bsXbtWpw8eRJ33nknmpqa4HK5YDQaYbVag6ax2WxwuVwAAJfLFRQ8He0dbddSWFgIi8WiDAkJCaF0m4j6oZCO+dx3333Kv8ePH4/MzEyMHDkSf/jDHxAeHt7jnetQUFCA/Px85WePx8MAIhrgunWq3Wq14nvf+x6OHz8Ou92O1tZWNDY2BtXU1tbCbrcDAOx2+xVnvzp+7qi5GpPJBLPZHDQQ0cDWrfBpbm7G//3f/yE+Ph7p6ekwGAwoKSlR2quqqlBdXQ2HwwEAcDgcqKioQF1dnVJTXFwMs9mM1NTU7nSFiAaYkL52/du//RseeOABjBw5EmfOnMHKlSuh0+kwa9YsWCwWzJ8/H/n5+YiJiYHZbMaiRYvgcDgwefJkAMD06dORmpqKOXPmYPXq1XC5XHjxxReRl5cHk8nUKwtIRP1TSOHz9ddfY9asWaivr8ewYcMwdepU7Nu3D8OGDQMAvPbaa9BqtZg5cya8Xi+ys7Px5ptvKtPrdDps2bIFubm5cDgciIyMxNy5c/Hyyy/37FIRUb+nERFRuxOh8ng8sFgscLvdPP5D1I+Esm3y3i4iUgXDh4hUwfAhIlUwfIhIFQwfIlIFw4eIVMHwISJVMHyISBUMHyJSBcOHiFTB8CEiVTB8iEgVDB8iUgXDh4hUwfAhIlUwfIhIFQwfIlIFw4eIVMHwISJVMHyISBUMHyJSBcOHiFQRcvicPn0aTz75JGJjYxEeHo5x48bh4MGDSruIYMWKFYiPj0d4eDiysrJw7NixoM9oaGjA7NmzYTabYbVaMX/+fDQ3N3d/aYhowAgpfC5cuIApU6bAYDBg27ZtOHr0KH71q19hyJAhSs3q1auxZs0arFu3DmVlZYiMjER2djZaWlqUmtmzZ6OyshLFxcXYsmULdu/ejQULFvTcUhFR/ycheOGFF2Tq1KnXbA8EAmK32+XVV19VxjU2NorJZJJ3331XRESOHj0qAOTAgQNKzbZt20Sj0cjp06c71Q+32y0AxO12h9J9IuploWybIe35fPTRR8jIyMBjjz2GuLg4pKWl4a233lLaT548CZfLhaysLGWcxWJBZmYmnE4nAMDpdMJqtSIjI0OpycrKglarRVlZ2VXn6/V64fF4ggYiGthCCp8TJ05g7dq1SE5Oxo4dO5Cbm4vnnnsOGzZsAAC4XC4AgM1mC5rOZrMpbS6XC3FxcUHter0eMTExSs13FRYWwmKxKENCQkIo3Saifiik8AkEApg4cSJeeeUVpKWlYcGCBXjmmWewbt263uofAKCgoABut1sZampqenV+RNT7Qgqf+Ph4pKamBo0bM2YMqqurAQB2ux0AUFtbG1RTW1urtNntdtTV1QW1+3w+NDQ0KDXfZTKZYDabgwYiGthCCp8pU6agqqoqaNyXX36JkSNHAgCSkpJgt9tRUlKitHs8HpSVlcHhcAAAHA4HGhsbUV5ertTs3LkTgUAAmZmZXV4QIhpgQjmSvX//ftHr9fKLX/xCjh07Ju+8845ERETI73//e6Vm1apVYrVa5cMPP5TDhw/LQw89JElJSXL58mWlZsaMGZKWliZlZWWyZ88eSU5OllmzZvXKEXUi6juhbJshhY+IyObNm2Xs2LFiMpkkJSVFfvOb3wS1BwIBWb58udhsNjGZTDJt2jSpqqoKqqmvr5dZs2ZJVFSUmM1mmTdvnjQ1NXW6Dwwfov4plG1TIyKi7r5X6DweDywWC9xuN4//EPUjoWybvLeLiFTB8CEiVTB8iEgVDB8iUgXDh4hUwfAhIlUwfIhIFQwfIlIFw4eIVMHwISJVMHyISBUMHyJSBcOHiFTB8CEiVTB8iEgVDB8iUgXDh4hUwfAhIlUwfIhIFQwfIlIFw4eIVMHwISJVMHyISBUMHyJSBcOHiFShV7sDXdHxklWPx6NyT4jo2zq2yc68CHlAhk99fT0AICEhQeWeENHVNDU1wWKxXLdmQIZPTEwMAKC6uvqGC3gz83g8SEhIQE1NzaB+Zz3XQ7v+sB5EBE1NTRg+fPgNawdk+Gi17YeqLBbLoP5j62A2m7kewPXQQe310NkdAh5wJiJVMHyISBUDMnxMJhNWrlwJk8mkdldUxfXQjuuh3UBbDxrpzDkxIqIeNiD3fIho4GP4EJEqGD5EpAqGDxGpYkCGzxtvvIFRo0YhLCwMmZmZ2L9/v9pd6jGFhYW4/fbbER0djbi4ODz88MOoqqoKqmlpaUFeXh5iY2MRFRWFmTNnora2NqimuroaOTk5iIiIQFxcHJYtWwafz9eXi9KjVq1aBY1GgyVLlijjBst6OH36NJ588knExsYiPDwc48aNw8GDB5V2EcGKFSsQHx+P8PBwZGVl4dixY0Gf0dDQgNmzZ8NsNsNqtWL+/Plobm7u60UJJgPMe++9J0ajUX77299KZWWlPPPMM2K1WqW2tlbtrvWI7Oxsefvtt+XIkSNy6NAh+dGPfiSJiYnS3Nys1Dz77LOSkJAgJSUlcvDgQZk8ebLccccdSrvP55OxY8dKVlaWfPrpp7J161YZOnSoFBQUqLFI3bZ//34ZNWqUjB8/XhYvXqyMHwzroaGhQUaOHClPP/20lJWVyYkTJ2THjh1y/PhxpWbVqlVisVjkgw8+kM8++0wefPBBSUpKksuXLys1M2bMkAkTJsi+ffvkb3/7m9x2220ya9YsNRZJMeDCZ9KkSZKXl6f87Pf7Zfjw4VJYWKhir3pPXV2dAJDS0lIREWlsbBSDwSBFRUVKzeeffy4AxOl0iojI1q1bRavVisvlUmrWrl0rZrNZvF5v3y5ANzU1NUlycrIUFxfL3XffrYTPYFkPL7zwgkydOvWa7YFAQOx2u7z66qvKuMbGRjGZTPLuu++KiMjRo0cFgBw4cECp2bZtm2g0Gjl9+nTvdf4GBtTXrtbWVpSXlyMrK0sZp9VqkZWVBafTqWLPeo/b7Qbwzc205eXlaGtrC1oHKSkpSExMVNaB0+nEuHHjYLPZlJrs7Gx4PB5UVlb2Ye+7Ly8vDzk5OUHLCwye9fDRRx8hIyMDjz32GOLi4pCWloa33npLaT958iRcLlfQerBYLMjMzAxaD1arFRkZGUpNVlYWtFotysrK+m5hvmNAhc/58+fh9/uD/pgAwGazweVyqdSr3hMIBLBkyRJMmTIFY8eOBQC4XC4YjUZYrdag2m+vA5fLddV11NE2ULz33nv45JNPUFhYeEXbYFkPJ06cwNq1a5GcnIwdO3YgNzcXzz33HDZs2ADgm+W43jbhcrkQFxcX1K7X6xETE6PqehiQd7UPFnl5eThy5Aj27Nmjdlf6XE1NDRYvXozi4mKEhYWp3R3VBAIBZGRk4JVXXgEApKWl4ciRI1i3bh3mzp2rcu+6Z0Dt+QwdOhQ6ne6KMxq1tbWw2+0q9ap3LFy4EFu2bMFf//pXjBgxQhlvt9vR2tqKxsbGoPpvrwO73X7VddTRNhCUl5ejrq4OEydOhF6vh16vR2lpKdasWQO9Xg+bzTYo1kN8fDxSU1ODxo0ZMwbV1dUAvlmO620TdrsddXV1Qe0+nw8NDQ2qrocBFT5GoxHp6ekoKSlRxgUCAZSUlMDhcKjYs54jIli4cCE2bdqEnTt3IikpKag9PT0dBoMhaB1UVVWhurpaWQcOhwMVFRVBf3DFxcUwm81X/CH3V9OmTUNFRQUOHTqkDBkZGZg9e7by78GwHqZMmXLFpRZffvklRo4cCQBISkqC3W4PWg8ejwdlZWVB66GxsRHl5eVKzc6dOxEIBJCZmdkHS3ENqh3q7qL33ntPTCaTrF+/Xo4ePSoLFiwQq9UadEZjIMvNzRWLxSK7du2Ss2fPKsOlS5eUmmeffVYSExNl586dcvDgQXE4HOJwOJT2jlPM06dPl0OHDsn27dtl2LBhA+oU89V8+2yXyOBYD/v37xe9Xi+/+MUv5NixY/LOO+9IRESE/P73v1dqVq1aJVarVT788EM5fPiwPPTQQ1c91Z6WliZlZWWyZ88eSU5O5qn2rviv//ovSUxMFKPRKJMmTZJ9+/ap3aUeA+Cqw9tvv63UXL58Wf71X/9VhgwZIhEREfLjH/9Yzp49G/Q5p06dkvvuu0/Cw8Nl6NCh8vzzz0tbW1sfL03P+m74DJb1sHnzZhk7dqyYTCZJSUmR3/zmN0HtgUBAli9fLjabTUwmk0ybNk2qqqqCaurr62XWrFkSFRUlZrNZ5s2bJ01NTX25GFfgIzWISBUD6pgPEd08GD5EpAqGDxGpguFDRKpg+BCRKhg+RKQKhg8RqYLhQ0SqYPgQkSoYPkSkCoYPEamC4UNEqvh/GD+Z27lAmJgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 300x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "-130.82398986816406"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "play(True)[-1]"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "第9章-策略梯度算法.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python [conda env:cuda117]",
   "language": "python",
   "name": "conda-env-cuda117-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
